This notebook is going to look at the mistakes made by the chosen model. Specifically, it's going to look at the log loss for each class on the validation set and produce a Hinton diagram for all the classes to show the confusion matrix for this model. Then, it will find the class performing worst and plot example images from this class after processing.
Aiming for it to be written in such a way that the model file chosen can just be swapped in, so we can try it on different models.
Doing the exact same thing as in the notebook on Holdout set testing, creating a set of predictions:
In [5]:
# copying imports from holdout notebook...
import pylearn2.utils
import pylearn2.config
import theano
import neukrill_net.dense_dataset
import neukrill_net.utils
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import holoviews as hl
%load_ext holoviews.ipython
import sklearn.metrics
In [9]:
cd ..
In [192]:
settings = neukrill_net.utils.Settings("settings.json")
run_settings = neukrill_net.utils.load_run_settings(
"run_settings/replicate_8aug.json", settings, force=True)
In [193]:
# loading the model
model = pylearn2.utils.serial.load(run_settings['pickle abspath'])
In [194]:
# loading the data
yaml_string = neukrill_net.utils.format_yaml(run_settings, settings)
proxd = pylearn2.config.yaml_parse.load(yaml_string, instantiate=False).keywords['dataset']
proxd.keywords['force'] = True
proxd.keywords['training_set_mode'] = 'validation'
dataset = pylearn2.config.yaml_parse._instantiate(proxd)
In [195]:
batch_size=500
while len(dataset.X)%batch_size != 0:
batch_size += 1
n_batches = int(len(dataset.X)/batch_size)
# set this batch size
model.set_batch_size(batch_size)
# compile Theano function
X = model.get_input_space().make_batch_theano()
Y = model.fprop(X)
f = theano.function([X],Y)
In [196]:
print(batch_size)
In [197]:
%%time
y = np.zeros((len(dataset.X),len(settings.classes)))
i=0
iterator = dataset.iterator(batch_size=batch_size,num_batches=n_batches,
mode='even_sequential')
for batch in iterator:
print(i)
y[i*batch_size:(i+1)*batch_size] = f(batch[0])
i+=1
In [198]:
labels = dataset.y
In [199]:
N = y.shape[0]
In [200]:
logloss = lambda x: -(1./N)*np.log(x[0][x[1]])
In [201]:
# iterate over all the images and make a list of log losses
ilabels = np.where(labels)[1]
In [203]:
losses = []
for r,l in zip(y,ilabels):
losses.append(logloss([r,l]))
losses = np.array(losses)
Getting an array of original image sizes:
In [204]:
sizes = []
for i in dataset.X:
sizes.append(np.sqrt(i.size))
sizes = np.array(sizes)
Then we can just make a scatter plot:
In [117]:
plt.scatter(sizes,losses)
plt.grid()
Doesn't appear to be a clear relationship between image size and log loss. We can look at the average log loss in a rolling average over the above plot and see where most of the log loss is:
In [146]:
np.hanning(50)*np.ones(50)
Out[146]:
In [175]:
ravg = np.zeros(450)
for i in range(450):
s = losses[(sizes > i-50) * (sizes < i+50)]
# apply window function
s = s*np.hanning(s.size)
if s.size != 0:
ravg[i] = np.mean(s)
else:
ravg[i] = 0
In [177]:
plt.plot(range(450),ravg)
Out[177]:
So it looks like we are losing more log loss per image at higher image sizes. However, most of log loss is going to be at the lower image sizes because there are more images there. If we do a rolling sum we'll see this:
In [178]:
rsum = np.zeros(450)
for i in range(450):
s = losses[(sizes > i-30) * (sizes < i+30)]
# apply window function
s = s*np.hanning(s.size)
if s.size != 0:
rsum[i] = np.sum(s)
else:
rsum[i] = 0
In [180]:
plt.plot(range(450),rsum)
Out[180]:
So, cumulatively we should probably worry more about the lower image sizes anyway.
In [191]:
import sklearn.linear_model
In [205]:
lreg = sklearn.linear_model.LinearRegression()
In [207]:
lreg.fit(sizes.reshape(-1,1),losses)
Out[207]:
In [214]:
plt.scatter(sizes,losses)
plt.plot(sizes,lreg.predict(sizes.reshape(-1,1)))
Out[214]:
In [216]:
plt.plot(sizes,lreg.predict(sizes.reshape(-1,1)))
Out[216]:
So there is a very small correlation between image size and log loss.
In [219]:
lreg.coef_[0]
Out[219]:
In [181]:
settings = neukrill_net.utils.Settings("settings.json")
run_settings = neukrill_net.utils.load_run_settings(
"run_settings/parallel_conv.json", settings, force=True)
In [182]:
model = pylearn2.utils.serial.load(run_settings['pickle abspath'])
In [183]:
# loading the data
yaml_string = neukrill_net.utils.format_yaml(run_settings, settings)
proxd = pylearn2.config.yaml_parse.load(yaml_string, instantiate=False).keywords['dataset']
proxd.keywords['force'] = True
proxd.keywords['training_set_mode'] = 'validation'
dataset = pylearn2.config.yaml_parse._instantiate(proxd)
In [185]:
batch_size=500
while len(dataset.X)%batch_size != 0:
batch_size += 1
n_batches = int(len(dataset.X)/batch_size)
# set this batch size
model.set_batch_size(batch_size)
# compile Theano function
X = model.get_input_space().make_batch_theano()
Y = model.fprop(X)
f = theano.function(X,Y)
In [186]:
%%time
y = np.zeros((len(dataset.X),len(settings.classes)))
i=0
iterator = dataset.iterator(batch_size=batch_size,num_batches=n_batches,
mode='even_sequential')
for batch in iterator:
print(i)
y[i*batch_size:(i+1)*batch_size] = f(batch[0],batch[1])
i+=1
In [187]:
losses = []
for r,l in zip(y,ilabels):
losses.append(logloss([r,l]))
losses = np.array(losses)
The scatter plot looks slightly better at the higher values:
In [188]:
plt.scatter(sizes,losses)
plt.grid()
In [189]:
ravg = np.zeros(450)
for i in range(450):
s = losses[(sizes > i-50) * (sizes < i+50)]
# apply window function
s = s*np.hanning(s.size)
if s.size != 0:
ravg[i] = np.mean(s)
else:
ravg[i] = 0
plt.plot(range(450),ravg)
Out[189]:
In [190]:
rsum = np.zeros(450)
for i in range(450):
s = losses[(sizes > i-30) * (sizes < i+30)]
# apply window function
s = s*np.hanning(s.size)
if s.size != 0:
rsum[i] = np.sum(s)
else:
rsum[i] = 0
plt.plot(range(450),rsum)
Out[190]:
It might do a little better, but it's difficult to see any kind of difference that would be significant.